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ABSTRACT " 

Methods and procedures were developed for monitoring 
and evaluating classrooms on critical dimensions derived from 
process-product research. The evaluation process for each of these 
student behaviors, related to improving academic achievement, is 
described. The dimensions include student engaged time, which is a 
product of allocated time and engagement rate; content coverage, 
including" prior learning and instructional overlap; and success rate, 
including such aspects of student academic success as success on 
daily work and success on topid tests. Procedures for evaluating the 
effectiveness of improvement strategies over time are described, A 
variables management strategy called the instructional improvement 
\ cycle was adopted to assist teachers and administrators in making 
\decisions. This strategy assumes that student achievement is more 
likely to increase if classrooms are at appropriate levels on student 
Engaged time, content coverage, and success rate. The necessity for 
fbrmal data collection in monitoring critical student behaviors and 
in\ meeting requirements of accountability and effective performance 
isXdiscussed, (DWH) 
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MONITORING AND EVALUATING THJX CRITICAL 
DIMENSIONS OF EFFECTIVE CLASSROOMS - 



Recent research has provided Bvldence that certain classroom condi- 
tions and processes are strongly linked to student achievement gains, 
especially In the elementary grades. However, In order to make u§e of 
this research,, practitioners must know what aspects of the classroom' are 
of particular Importance, how to monitor and improve those aspects, and 
whether the selected Improvement strategies are having an effect on class- 
room learning. 

Research for Better Schools (RBS) has developed ah Instructional 
improvement approach addressing these Issues. RBS has previously reported 
on the selection of three critical student ^behaviors which teachers and 
administrators can monitor throughout the school year and a variables 
management strategy by which educators can address these variables 
(Helms, 1980). RBS has also previously reported on Instruments and 
procedures which can be used in the monitoring process (Hultt, Caldwell) 
Traver, & Graeber, 1981; Segars,^ Caldwell, Graeber, & Hultt, J981), and 
Strategies selected from research which can be used to improve. the three 
critical studenc behaviors (Caldwell, Hultt, & French, 1981; Caldwell, 
' Traver, Segars, French, & Hultt, 1981). The purpose of this paper is ^o 

r^eport on the development of methods and procedures that can be used to 
-evaluate the effects of selected Improvement strategies. 

The next section provides an overview of the variables management 
strategy called the instructional Improvement cycle. The following three 
sections describe the evaluation process for each of the three student 
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behaviors or variables identified by RBS as being especially important:' 
student engaged time, content coverage, and success. The final sect3oi\^of 
the paper summarizes RBS's experiences in facilitating use of these 
methods anS procedures. 

Variables Management Strategy ^ 

In order to facilitate teachers' and administrators' utilization of 
research for making Uecisions pertinent to Improving classroom instru^ 
tion^ RBS has adopted a simple variables management strategy that can be 
used in making classroom modifications based on the needs of individual 
classroom situations. This strategy, which RBS has labeled a "four-phase, 
instructional Improvement cycle" (Figure 1), calls for collection of 
classroom data, comparison of classroom data with a data base that relates 
classroojn conditions /processes to students' achievement, making decisions - 
about what and how to modify" instruction, and implementation of planned 
modifications. Since the improvement cycle is iterative^^gubsequent 
collections of classroom data permit practitioners to evaluate the effec- 
tiveness of their classroom modifications. On the basis of such evalua- 
tion, they are in a position to decide the continuation, revision, or 
discontinuation of such modifications. 



Figure 1 about here 



There are three separate monitoring/evaluation questions which arise 
in the use of improvement cycle. First, upon initial use of the cycle. 
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there is the question of current status of an individual student behavior 
In a particular classroom at a specific moment in time* Second, if a 
djgcision is made to modify classroom practice an^ an improvement strategy 
is implemented, then the subsequent use of the cycle raises an additional 
question as to the effectiveness of that particular strategy at a specific 

P9int in time. However, there is a third question of interest: What are 

. c 

the cumulative effects*of -using the improvement cycle over the school 
year? While the data collected during a single observation may provide an~ 
accurate picture of" the classroom at that point in time, numerous studies 
(e.g.. Medley & Mitzel, 1963'; Rowley, 1978) have shown that single esti- 
mates do not provide valid and reliable information. about the entire 
school year. Valid and reliable information is needed In order to eval- 

0 

uate tKe cumulative effects of using the improvement cycle. There are two ^ 

' aspects of these effects to be considered: (1) the impact of the 

implemented strategies on the student behavior and (2) the relationship of 

the student behavior to achievement as predicted from the research 

findings. The remainder of this paper will focus on instruments and 

pfrpcedures which address this third question. 
«» 

Student Engaged "Time 

■ t 

Student engaged time refers to the number of minutes that students 
are actively working on academic content such as reading/language arts or 
mathematics. This variable is the product of two other time variables: 
allocated time, the amount of time provided by the teacher for instruc- 
tion, and engagement rate, the percentage of time students are actually 
working on academic tasks For exampj_e^ __i_f sjtudeats .are aetuaily^working- — 



on academic tasks 50 percent "of the time during a 60-minute period, then 
«the student engaged time is 30 minutes. 

Student engaged time has been shown to be significantly related to 
student achievement in numerous studies.- RBS has reanalyzed the data from 
two of chose studies (Stallings & Kaskowitz, 1974; Fisher, Marliave, 
Cahen, Dishaw, Moore, & Berliner, 1978) so that it can be used to heJp 
teachers and administrators identify opportunities for improvement in 
• their classrooms. Huitt and. Rim (1980) provide a complete discussion of 
the reanalysis procedure. 

The results of the reanalysis (Table 1) indicate whether given 
level's of student engaged time were associated with achievement below, at, 
% or above the expected level (i.e., residual gains that were negative, 
zero, or positive). Specific procedures for generating these zones are. 
described in Huitt and Rim (1980). In general, the data indicate that 
mpre student engaged time is better with the exception of third grade 
reading/language arts and first grade mathematics. In these cases, the 
relationship between eoigaged time and achievement was curvilinear. That 
is, as engaged time increased, achievement increased only up to a point 
and then began to decrease. 

Table 1 about here 



Teathers and administrators have used the data collection instruments 
described in Huitt et al. (1981) and the informaticjn .described in Table 1 
to make decisions about whether changes in the use of classroom time are 
needed. If an opportunity to i mprove^ e xists, strategies designed to 



impact, student engaged time can then be selected and implemented. The 
effectiveness* of these strategies can then be evaluated hy collecting a 
second round of data and comparing It, to* the original level of stude\^t 
engaged time. 

However, procedures for evaluating the status of student engaged time 

acjoss the year^are still an issue. RBS has developed observation records 

(Figure 1) which educators can use to record the time data collected over 

the school year. The vertical axis represents student fengaged time in 

f 

minutes per day and the horizontal axis represents the months of the 
school year. The vertical axis is, marked into three zones: the better 
than expected level of achievement zone is white, the at expected level of 
achievement zone is shaded, and the below expected level of achievement 
zone is slashed. In this example, a third grade mathematics classroom has 
been observed once a month over the course of the school year. Student 
engaged times for the first three^,9bservations were in the "below" zone, 
and the. remainder were in the "at" zone. If-rfhe goal were ,to move into 
the ''at" zone,' then it would probably be reasonable to conclude that the 
use of the improvement cycle was successful. However, it is also 
necessary to consider seasonal patterns in student engaged time throughout 
the year. For example, Evertson and Veldman (1981) have shown that 
engaged time is generally higher in the middle of the year than at the 
beginning^or end. Therefore, it is probably necessary to monitor the 
situation closely next year to determine whether the higher student 
engaged time is simply an indi^cator of the time of the year. ' 



' Figure 2 about here 

' ■■ ' 

0 * 

Content Coverage 

Content coverage refers to the concepts and skills that the student 
actually covers during the course of instruction. There are two important 
aspects of this variable: the student covers content that 1) is necessary 
for new learning (prior learning^ and 2) is relevant to the content tested 
(instructional overlap) • 

Research suggests that both aspects pf content coverage are signifi- 
cantly related to achievement. With respect to prior learning, Bloom 
(1976) estimates that 60 to 80 percent of the difference in student- ^ 
achievement test scores is due to differences in student's past learning. 
With respect to instructional overlap, RBS has reanalyzed data frofa the 
Instructional Dimensions Study /Codley b Leinhardt, 1980) in the same way 
as previously described for. student engaged time. Again, in general, more 
is better^ though that is not always so (Table 2). 



Table 2 about here . 
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In order to monitor content coverage, it is first necessary to plan 
instruction for the year by examining prior learning and instructional 
overlap. Prior learning is analyzed both in terms of the students 
entering level of achievement and In terms of their specific strengths and 
-.weaknesses.. „In8txuctijpml-,El^M jre est^^^^ away as to 



address these prior learning concerns and, at the same time, to obtain .a 
desired instructional overlap. If these plans are recorded on a school 
'year planning guide such as is shovm in Figure 3,^ then monitoring content 
coverage- is simply a matfter of recording the date each, topic is completed 
^rtd comparing the number of days', spent on that topic to the number of days 
planned. If students spent more time than was planned on a topic, the 
teacher may implement strategies to increase the*-p.ace of instruction on,„ 
the next unit. The success of the implemented strategies is then ev^I- 
uated by again comparing the number of days used to .the number of days 
planned. Evaluating the status of coverage over time is accDmplished by 
comparing the total number of days used thus far in the school year to the 
total number of days planned 'for covering those topics. 



Figure 3 about here 



The school year planning guide facilitates the making of short-term 

unit or topic plans. These are different from the usual weekly lesson 

plans in that planning is done in terms of units or topics gather than 

days or weeks. In addition, the objectives and skills covered In these ^ 

units' are in turn tied to the curriculum, topics on the test, and prior 

learning strengths and weaknesses through the school year planning guide. 

By making these kinds of plans, teachers are able to deter?nine quite early 

. { 1 
whether students are actually, covering content as planned. They can thus 

I 

make adjustments throughout the school year rather than waiting until' just 
before the test or the end of the year. 



Success 

Success- refers to students* performance on academic work. Ther6 are 

two Important aspects of student success that should be monitored^ , 

success on dally work. Including both new and rdvlew content and success 

/ 

on unit or topic tests. 

Several theories of Instruction feature the importance of student 
success for achievement (Bloom, 1976; Skinner, 1968). These theories are 
further supported by research findings indicating that both success on 

dally work (Fisher, et^al., 1978, 1979) and success on unit tests (Bloom, 

r ' ' * ' ^' J -1 

3 976) are significantly related to achievement. With respect to daxJy 

succejss, iTi Phase III-B of the Beginning Teachdr Evaluation Study (19780, 
Fisher and his colleagues report that students who completed more than 
■half of their academic tasks at. a high, level of success -(l^.e., no errors 
or only careless 'errors) had higher gains in achievement in reading and 
mathematics. Marllave and Fllby (1980) suggest in addition that monitor- 
ing students' success on dally tasks is one way to ascertain whether 
students have sufficient prior' leatnlng for beginning new academic tasks. 
With respect to success on tests, research on mastery learning indicates 
that students who are successful (80-85 percent correct) on uni,t err topic 
tests score higher on end-of-yeaf achievement tests (Bloom, 1976). Thus? 
the research li^lcates that it is Itnportapt for teachers to monitor 
success on both dally work and units of Instruction. 

Most teachers already monitor students' daily work by grading assign- 
mehta and recording scores in a grade book. However, mos^ students are 
probably not meeting standards of success suggested by research. Teaqhers 
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can analyze the data collected each day 'on students' success In order ta^ 

\ * * ^ ' • . ' ' ' * . ' 

decide whether daily success can be'dmproved^ Strategies for impr:oving 

student success then may be implemented. In evaluating th'e^ef ferfts of 

these strategies, It may be helpful. to .look at 'several written assignments 

(e.g.*, 20 workbook pages) at f time. For example, the number'of students 

working at a high level of success on at least half of the assignments may 

be recorded on the schooL year planning guide (Figure 3) at^ the timq of 

testing. If the number of students working at a high success level on at 

least half o^f the assignments has increased, then the implemented , 

strategies for improving daily success might b'e considered effective. By 

keeping tliis record across the school year ^. teachers and administrators 

can. assess the status of daily success over time. ' , . 

Most teachers monitor students' suQcess on units or topics by giving , 

tests periodically. Teachers may collect data on students' success by 

indicating whether students have mastered each concep.t or skilT tested. 

Tf students have not reached a satisfactory level of mastery (usually 

80-85 percent); then remediation strategies may be implemented. The 

' \ ... 

success of these strategies may be m^easured'by retesting students who did 

not reach the mastery ci;iterion previoasly. In order to evaluate the 
status of mastery throughout the year, records of student progress over 
time must be kept (Figure 4)*. Students' names are written on the side of 
the c^hart, and the topics, or objectives covered by unit tests are indi-^ 
cated at the to/ of the chart. The date on which each student demon- 
strates mastery is recorded for that student in^the column for that unit 
or topic. Using such a chart, it is easy to see 'exactly which students 
'have mastered which units or tqpicsi ^ • • 
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^Figure 4 about here 
Conclusion 



In summary, research. has shown that achievement is increased when 
students have adequate academic preparatioi^ for learnir*g new content and 
are involved with and* successful on content that is ^related to the 
!^ichlevement test. Jt is our Contention that student achieyement will be 
maximized' when teachers and prirtcipals monitor students' involvement, 
^coverage, and succfe&s; develop strategies- to improve- these behaviors when 
appropriate; ^nd evaluate the ef f ectivenesK^f those strategies .over time. 

Instruments and ^procedures for monitoring, as well as strategies for ^ 
improving, these critical student behaviors have been described in 
previous papeifs. RBS has adopted a variables management strategy which 
facilitates teachers' and' administrators ' in making 'decisions regarding' 
thesQ variables, rrhis-paper , 'in- turr\^, has described p^rocedures for » 
ev^luatinf^ the Effectiveness of improvement strategies over timeT* These 
same procedures can also be used to evaluate the status of the student^ 
behaviors, during f he, entire school year. .... 

Our experience With attempting to facili^te monitoring and eValp- . 
ation of these three critical student behaviors, indicates that. if d formal 
process is .to be^ carried out, then those. ill Jthe highest l^.vels. of ^ 
authority must desire to see^the results. In general, it seems that 
principals and teachers are satisfied with jnore informal' data. Tea-che^s 

] ' • I ' ' V • , 

report that after they become aware 'of the impprtaUce of time, content 
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^cDWfage, and success, they are more likely to start lessons on time., to 
look around the room to see that everyone is busy, to cover test-related 
content, and to look for ways for all students to be successful. However, 
they are often more concerned-with immediate activities than with 

long-term assessment. Principals, in turn, are likely to accept teachers' 

""^'[ ' ' \ < 

stat^pentG of; classroom behaviour rather than to monito^^ that behavior in a 

systematic way.^ It seems that formal monitoring becomes Xpmmonplace only 

when the superintendent or other district office supervisor Vants to see 

the data or when the 'principal uses the data as part of an evaluation 

' ' ' ' ^ ^ \ . 

system. \ 

In evaluating , the cumulative effects of use of. the improvement cycle, 
we are concerned primarily with comparing the statys of the critical 
^student behaviors to the research findings, although thsre is some concern 
with demonstrating improvement or change '-/here improvement strategies have 
been implemented. However, the question is often raisedl: Is formal data 
collection, necessary in order to produce change? Based on anecdotal 
evidence from principals and teachers, it is likely that awareness and 
training, in formal data collection are enougTi to stimulate practitioners 
to informally monitor. the critical student behaviors and to select . 
strategies to impact those behaviors as needed. However, documenting 
whether change has actually taken place (i»e., whether strategies have 
been effective), as well as documenting the status of critical student „ 
behaviors, requires more formal methodology. As_ requirements of accounta- 
bility and effective performance become more severe, there may be more 
necessity for such documentation. . . ' 



This paper has discussed methods and procedures that can be used by 
administrators and teachers to monitor and evaluate three critical dlnen- 
sions of classrooms: student engaged time, content coverage, and student 
success. Each of these aspects of the classroom is important in relation- 
ship to improving achievement. By monitoring, these dimensions and imple- 
menting strategies to improve them, and evaluating the effectiveness of 
those strategies over time, administrators and teachers can work together 
to improve instruction and student achievement. 
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Table 1 

Relationship of Student Engaged Time in Reading/Language Arts ' . 
and Mathematics to Student Achi^^^^ 



Readina/Lanauaqe Arts 


Below 




Above 


Grade 1 ^ 


, ,40-110 


110-130 


130-210 


3^ ■ ■ 


45-90 


90-115 


115-170* 


5' 


40-80 

* 


■ 80-90 


90-135 


Mathematics 


* 






.Grade 1 ^ 


•5-35 


35-45 


45-140^ 


■ 3*' 


10-45 


45-60 


60-100 


5' - 




Range = 15-45° 





♦ Student engaged time beyond this point was not positively 
related to student achievement. 

° Not significantly related to student achievement. 

t Data Source: Staliings and Kaskowltz, Follow Through Evaluation Study, 
A Data Source: Fisher et al., Beginning Teacher Evaluation Study, Phase 
1978. 
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Table 2 



Relationship- of In'stmctionarOverlap in Reading/Language Arts 
and Mathematics to Student Achievement 



Below At Above 



Reading/Language Arts 



\ 

\ 



Grade 0-55 55-70 70-100 

3-^ 0-45 45-75 75-100 



Mathematics 

Grade 1 0-35 " 35-40 40-90* 

3 ^ 10-50 ' 50-60 60-100 



* Jnstructional overlap beyond this point was not positively 
related to student achievement. 



t Data Source: Cooley and Lelnhardt, Instructional Dimensions Study, 
1980. 
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Collecting Information 

on Classroom 
Conditions/Processes 



fhaselt 



Comparing Data and 
Identifying Improvement 
Opportunities 







Classroom 







Phase IV 



Implementing and 
Monitoring 
Modifications 



Selecting and Preparing 
Classroom Modifications 



Phacelll 



Figure 1.. Four-Phase Instructional Improvement Cycle«f 
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MATHEMATICS OeSERVATIOH RECORD 



GRADE 

3 



s ^ 
e 5* 



70 



60 



i 



50 V 



I 



40 ;^ 



30 



I 



20 



Sept 



Oct Nov 



Dec Jan Feb Mar 

DATE OF OBSERVATIOW 



Apr 




Hay June 



EXPECTED tEVEt OF ACHIEVEMENT 
Qawve ^AT 0»EtOW 



* DATA SOURCE: Stalllnss k K^aakowltz (1974) 



Figure 2. Example of Completed;- Observation Record 
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SCHefOL YEAR PLANNING GUIDE 
Mathematics - Grade 4 , . * 


• -"^^^^OIl^LUM 


MATERIALS 


, TOPieS-ON 
CURRENT TEST 


PRIOR 

1 ^ i 111 xir* 

LEARNING 


DAYS 
NEEDED 


DATE 

.^.--^CCESS 


1 WHOLE NUMBERS 


NUMERATION/ . 

PLACE VALUE 
Place value 

to million 


Houghton Mifflin 
pp. 22-29 


Place value to 
' ten thousands 
Renaming numbers 




• 

3 




Roman numer- 
als to 100 


Houghton Mifflin 
p. 44 

Supplementary 
workbook pp. 4-5 


• 


♦ 


2 


^-ii ■ 


ADDITION & 
SUBTRACTION 
Facts 


Houghton Mifflin 
pp. 2-16 






4 




Regrouping up 
to ten thou- 
sands place 


Houghton Mifflin 

pp. 47-69 
Supplementary 

workbook pp 15-18 


Up tp 5 digits, 
vertical and 
horizontal 
formats 




--^4rv, ' 





Total 150 days 



Figure 3. SampJe School Year Planning Guide 
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